With improvement on performance of a computer, realization of a storage apparatus having a large capacity, expansion of a communication infrastructure and other progresses, a service or the like has been coming into wide spread use that maintains a data base of a large amount of pictures and delivers a necessary picture when a necessity arises. Since a technology for digitizing a picture has progressed, an individual can build a data base of motion pictures.
When a data base of motion pictures has been built, then how to retrieve a desired picture therefrom becomes a problem. The present invention relates to an improvement on such a technology for retrieval of a motion picture.
The term “motion picture” used in the present specification includes both types of motion picture data not processed and constituted of RGB (Red, Green and Blue) signals and motion picture data digitized and coded according to a motion picture compression format as is in MPEG (Moving Picture Experts Group) or the like.
The term “scene” means a constituent unit of a motion picture including a consecutive plural picture frames and for example, of a motion picture obtained in one time shooting from the start to the end of recording or a motion picture for which editing has been finished, a length of the motion picture partitioned with two edit points at both ends thereof. An edit point includes, in a case, special effects such as fading and a dissolve. Furthermore, one scene may include plural scenes.
The term “a scene change point” means a position on the time axis at which two scenes are switched over therebetween and a leading frame directly after a scene is switched is referred to as “a scene change frame.”
The term “a key frame” means a distinctive frame utilized in retrieval or the like, representing a scene or all of a motion picture.
A motion picture data is “contents” constituted of image information changing along the time axis. Motion picture data becomes an object of retrieval in the entirety. Therefore, the simplest method for retrieval of a desired motion picture is to select a desired motion picture by seeing all of a motion picture. However, it takes a time to see all of a motion picture. Therefore, when the number of motion pictures registered at a data base increases, it takes an extremely long time to take out a desired motion picture. Accordingly, this method is unrealistic.
In a generally adopted method for efficient retrieval of a motion picture, various kinds of information for retrieval (retrieval information) representing contents of respective motion pictures are prepared so as to be ready for use in retrieval of the respective motion pictures and retrieval is performed on the basis of the retrieval information.
Information used as retrieval information includes information attached to a picture (attached information), information expressing characteristics of a picture (characteristic information) and information expressing an outline of a picture (outline information). The attached information includes a title or a descriptive text prepared accompanying a motion picture. The characteristic information includes information expressing characteristics of a picture such as a frequency of a luminance value derived from a picture signal, motion information between frames, or the like. The outline information includes information of scene change points, key frames, or the like, which are determined by characteristic information of a picture or an input from the outside.
Of the above-described information, the characteristic information is especially convenient and important. For example, a key frame representing a motion picture can be selected by using characteristic information. The characteristic information is easily digitized and if a characteristic of a desired motion picture is digitized in the form of characteristic information, it can be easy to extract a motion picture having a characteristic coinciding with a requested characteristic information. In the following description, the term “retrieval information” means this characteristic information.
In general, a motion picture includes plural scenes. When a retriever retrieves a motion picture, the retrieval is performed in order to find out a desired frame or a desired scene included in a motion picture or one of a plurality of motion pictures which are an object for retrieval. A fundamental method for finding out a desired frame or a scene in a motion picture using retrieval information is to follow a procedure in which retrieval information relating to frames or retrieval information relating to scenes is obtained in advance and a frame or a scene is selected on the basis of such retrieval information.
An apparatus and process are disclosed in Japanese Patent Laying-Open No. 9-284702 in connection with this point in which a total sum of motion vectors of each frame and a continuity of a plurality of small regions constituting a frame, in a frame and between frames are adopted as feature of a frame for detection of a scene change point.
Furthermore, a control method is disclosed in Japanese Patent Laying-Open No. 7-38842 in a case where the average of motion vectors in magnitude and color histograms of pictures in each scene are used as a feature of the scene and a representative picture of each scene is displayed as a picture index.
If an apparatus for storing information for retrieving a motion picture (a motion picture retrieval information storage apparatus) and an apparatus for retrieving a motion picture (a motion picture retrieving apparatus) are fabricated, taking into consideration such prior art techniques, the apparatuses will be as shown below.
Referring to FIG. 1, a prior art motion picture retrieval information storage apparatus which is assumed includes: an analysis section 601 for analyzing inputted motion picture data to divide the data into scenes and output information expressing a structure of a motion picture such as scene dividing positions (scene change); a retrieval information generating section 602 for a frame unit, connected so as to receive an output of analysis section 601 and inputted motion picture data and for generating and outputting summary information of a picture such as a total sum of motion vectors corresponding to each of frames in a scene as retrieval information on the basis of motion picture structure information and motion picture data outputted from analysis section 601; a retrieval information generating section 603 for a scene unit, connected so as to receive an output of retrieval information generating section 602 for a frame unit and generating and outputting summary information such as the average value of motion vectors in magnitude for all of a scene as retrieval information on the basis of summary information of a frame unit outputted from retrieval information generating section 602 for a frame unit; a retrieval information generating section 604 for a motion picture for arranging, into a prescribed format, motion picture structure information outputted from analysis section 601, retrieval information of each frame outputted from retrieval information generating section 602 for a frame unit and retrieval information for all of a scene outputted from retrieval information generating section 603 for a scene unit to generate and output retrieval information corresponding to a motion picture; and a storage section 605 for storing original motion picture and the retrieval information corresponding to a motion picture outputted from retrieval information generating section 604 for a motion picture, in a state of both being related to each other, into a storage medium 606.
Note that one frame can be considered to be a unit of a motion picture obtained by temporal division of the motion picture. Hence, a frame can be called as one example of a temporally divided block.
In this specification, a notation of “retrieval information of a frame unit” is used for expressing retrieval information provided with a frame as a unit. Similarly, a notation of “retrieval information of a scene unit” is used for expressing retrieval information provided with a scene as a unit.
Referring to FIG. 2, a prior art motion picture retrieving apparatus which is assumed includes: a read-out section 701 for reading out motion picture retrieval information and motion picture data from storage medium 606; a retrieval information managing section 702 for holding each motion picture retrieval information supplied from read-out section 701 and receiving a request for retrieval information to output retrieval information of a scene unit and retrieval information of a frame unit together with structure information of a motion picture; a retrieval executing section 703 for a scene unit for receiving a request for retrieval with a scene as a unit to request retrieval information of a scene unit and motion picture structure information from retrieval information managing section 702 and detecting a scene meeting the retrieval request on the basis of the retrieval information of a scene unit and the motion picture structure information of a scene received to output information on the detected scene; a retrieval executing section 704 for a frame unit for receiving a request for retrieval with a frame as a unit to request retrieval information of a frame unit in a scene and motion picture structure information from retrieval information managing section 702 and detecting a scene or a frame meeting the retrieval request on the basis of the retrieval information of a frame unit in a scene and the motion picture structure information received to output information on the detected scene or frame; a data managing section 705 for taking out and outputting picture data corresponding to a detected scene or frame through read-out section 701 on the basis of information on the detected scene or information on the detected frame, outputted from retrieval executing section 703 for a scene unit and retrieval executing section 704 for a frame unit; and a retrieval control section 706 for responding to a retrieval instruction given by an operator to provide a retrieval request to retrieval executing section 703 for a scene unit or retrieval executing section 704 for a frame unit, and for repeating a process displaying motion picture data provided from data managing section 705 as a retrieval result to output the retrieval result at the end of retrieval.
The apparatus shown in FIG. 1 is considered to operate as follows: When motion picture data is inputted, analysis section 601 divides motion picture data into scene units. Analysis section 601 further outputs motion picture structure information.
Retrieval information generating section 602 for a frame unit generates summary information corresponding to a frame in a scene on the basis of motion picture structure information outputted from analysis section 601 and motion picture data. Retrieval information generating section 602 for frame unit further outputs the summary information thus generated to retrieval information generating section 603 for a scene unit and retrieval information generating section 604 for a motion picture as retrieval information.
Retrieval information generating section 603 for a scene unit generates summary information for all of a scene on the basis of the summary information of a frame unit provided from retrieval information generating section 602 for a frame unit. Retrieval information generating section 603 for a scene unit further provides the summary information thus generated to retrieval information generating section 604 for a motion picture as retrieval information.
Retrieval information generating section 604 for a motion picture receives motion picture structure information, retrieval information of a frame unit and retrieval information for all of a scene, from analysis section 601, retrieval information generating section 602 for a frame unit and retrieval information generating section 603 for a scene unit, respectively, and generates retrieval information corresponding to a motion picture, arranging the information into a prescribed format. Retrieval information generating section 604 for a motion picture provides the retrieval information corresponding to a motion picture thus generated to storage section 605.
Storage section 605 stores original motion picture data and the retrieval information provided from retrieval information generating section 604 for a motion picture into storage medium 606. At this time, storage section 605 attaches information indicating that the motion picture data and the retrieval information, to be both stored there, correspond with each other, to one or both of the motion picture data and the retrieval information; or stores the information of the correspondence separately from the motion picture data and the retrieval information.
Referring to FIG. 2, it is assumed that the prior art retrieving apparatus operates in the following way: At the start of retrieval, an operator gives a retrieval instruction to retrieval control section 706. Retrieval control section 706 selects retrieval of a scene unit or retrieval of a frame unit in response to the retrieval instruction.
It is assumed that, for example, retrieval of a scene is selected. Then, retrieval control section 706 provides a retrieval request to retrieval executing section 703 for a scene unit. Retrieval executing section 703 for a scene unit responds to this request to request retrieval information of a scene unit and motion picture structure information from retrieval information managing section 702 and receives the retrieval information of a scene unit and motion picture structure information outputted from retrieval information managing section 702.
Retrieval executing section 703 for a scene unit detects a scene meeting the retrieval request provided from retrieval control section 706 on the basis of the retrieval information and the motion picture structure information of a scene unit. Retrieval executing section 703 for a scene unit outputs information on the detected scene to data managing section 705.
Data managing section 705 reads out corresponding picture data from storage medium 606 through read-out section 701 on the basis of information on a scene provided from retrieval executing section 703 for a scene unit to provide the picture data to retrieval control section 706.
Retrieval control section 706 displays the picture data. The operator sees the displayed picture data to determine a next retrieval policy and gives a next retrieval instruction to retrieval control section 706. The similar process is repeated hereinafter.
A picture that the operator desires to take out from the data base is taken out in such a way, and then the retrieval finishes. A retrieval result is outputted from retrieval control section 706.
In the prior art, retrieval information of a scene is generated on the basis of retrieval information of frame units included in the scene. This is because of general understanding that the minimum unit constituting a scene is a frame.
Retrieval information of a scene is generated on the basis of retrieval information of frame units in such a way and then spatial features in each frame obtained by splitting the frame into small regions (retrieval information of a screen-split block units) can be used as retrieval information. However, a problem has been encountered since in a prior art apparatus, spatial features of a picture over all of a scene having a length on the time axis cannot be handled. In the present specification, such spatial features of a picture over all of a scene is referred to as “retrieval information of a spatially split block unit.”
Consider, for example, a case where a scene is taken out that has a feature that an object in a vigorous motion is displayed in a middle portion of a screen. In the prior art, it was not able to take out such a scene from retrieval information of a scene unit but a necessity arose for retrieval using retrieval information attached to each of frames included in a scene. For this reason, in the prior art, such retrieval was performed with a very low efficiency.
The present invention has been made in light of the above-described problem and it is an object of the present invention to provide a motion picture retrieval information storage apparatus and a motion picture retrieving apparatus capable of efficiently performing retrieval of a motion picture using a new index expressing a feature of a scene with correctness.
It is another object of the present invention to provide a motion picture retrieval information storage apparatus and a motion picture retrieving apparatus capable of efficiently performing retrieval of a motion picture by extracting not only a feature of a scene by dividing a scene into blocks on the time axis but also another feature by splitting the scene only spatially without dividing the scene on the time axis.
It is still another object of the present invention to provide a motion picture retrieval information storage apparatus and a motion picture retrieving apparatus capable of efficiently performing retrieval of a motion picture using a new index expressing a feature of a scene with correctness, with an small amount of information for the retrieval.